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ABSTRACT 

This paper describes the implementation of problem-based learning (PEL) principles in 
an undergraduate IT sendee management course, followed by the results about student 
satisfaction and work performance. The results indicate the students’ general 
satisfaction with the course implementation, as well as some challenges regarding the 
self-assessment and peer assessment of their work. The findings also reveal the 
students ’ better work performance in project results than in traditional knowledge tests, 
which reinforces the indications about their positive attitudes towards the interactive 
PBL environment. The cluster analyses identified seven different patterns in student 
behaviour regarding course performance. Findings from this study could be used by 
both researchers and practitioners in their efforts to create or further research a 
stimulating, interactive learning environment based on PBL that improves the 
preparation of students for their future workplaces. 
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INTRODUCTION AND MOTIVATION 


Current predictions about future trends in the demand for information and communication 
technology (ICT) professionals in Europe present three different scenarios for the period until 
2020 (Gareis et al., 2014). In all three cases, the demand potential exceeds the predicted 
number of ICT graduates. Therefore, the graduate labour market supply and demand has 
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emerged as an important issue for research (Atfield & Purcell, 2010; Mourshed, Patel, & 
Suder, 2013) and has found its place in policy documents as well. Preparing graduates in the 
ICT field for a successful professional career requires continuous improvement in the 
educational process and curriculum design (Ali & Aliyar, 2012; Pilgrim, 2013), which 
includes new teaching methods and collaboration with employers. Innovative teaching 
methods, such as work-ready learning activities (Costley & Dikerdem, 2012; Litchfield & 
Sixsmith, 2010; Shukla, Costley, & Inceoglu, 2011), experience-based learning (Matsuo, 
Wong, & Lai, 2008), practice-based learning (Hynes, Costin, & Birdthistle, 2011), 
cooperative education programmes (Coll, Zegward, & Hodges, 2002), problem-based 
learning (PBL) (Intayoad, 2014) and so on, are increasingly finding their place in formal 
education. 

The general aim of this paper is to provide a framework for organising and implementing a 
curriculum for an IT service management course, respecting the principles of (PBL) and 
related practices. The framework encompasses the definitions of all the main curriculum 
elements, including curriculum aims, intended learning outcomes, course content, learning 
and teaching methods and assessment (McKimm, 2007). The description of implemented 
framework is followed by the analysis of student satisfaction with certain curriculum elements 
implemented according to PBL principles, amended with the analysis of different patterns in 
students’ behaviour regarding the assessment of their work performance during the course. In 
line with these research goals, the authors have proposed two research questions: 

1) How do students assess particular aspects of this set learning environment, 
organised according to problem-based learning principles? 

2) Is it possible to identify specific patterns in students’ work performance in the 
established problem-based learning environment? 

First, the paper brings short theoretical introduction on PBL principles, followed with the 
description of their implementation in IT service management course. In the Methodology 
chapter, we described two different methods used to answer the research questions - semantic 
differential scale for the assessment of different aspects of IT service management course 
organized according to PBL learning environment and cluster analysis for the identification of 
different patterns in students’ work performance. Paper is concluded with the extensive 
Discussion of the results and implications for further research. 


THEORETICAL BACKGROUND 

This chapter contributes to the understanding of problem-based learning principles and 
provides detail explanation of IT service management curriculum elements. 
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Problem-based learning 

Problem-based learning is a student-centred instructional approach (Popescu, 2012) that 
involves valorising the problems from the “real world” in the educational process to facilitate 
the development of students’ critical thinking and problem-solving abilities and the 
assimilation of the fundamental concepts for different academic disciplines (Draghicescu, 
Petrescu, Cristea, Gorghiu, & Gorghiu, 2014). The PBL process sets goals for students to 
plan, communicate their ideas, gather information and implement and evaluate projects with 
real-world applications (Dominguez & Jaime, 2010) and are thus critical for learners in higher 
education to acquire genuine experiences (Heo, Lim, & Kim, 2010). Some authors describe 
PBL as the shift from traditional educational approaches to innovative ones that encourage 
lifelong, collaborative, student-centred and self-regulated learning (Baturay & Bay, 2010) and 
as such, contribute to the development of the students’ necessary skills for their future 
workplaces. The relevant literature has also recognised the positive influence of PBL methods 
on the development of students’ generic and transferable skills to fulfil the needs of the 21 st - 
century job market, such as problem solving, creativity, teamwork and critical thinking, 
among others (Ersoy & Baser, 2014; Hung, Hwang, & Huang, 2012; Musa, Mufti, Latiff, & 
Amin, 2011, 2012; §endag & Ferhan Odaba§i, 2009). From the students’ point of view, PBL 
contributes to improvements in terms of learning, motivation, enjoyment, involvement, 
teamwork quality and overall satisfaction, but it also increases their study time and effort 
compared with traditional learning (Popescu, 2012). Since PBL tasks refer to real-world 
problems, they are usually too complex for individual work and imply co-operation among 
students in the form of teamwork. In this manner, PBL often integrates co-operative learning 
(Heo et al., 2010) and can thus be described as an interactive learning environment based on 
real-world problem solving. 

Implementation of PBL in IT service management course 

To successfully achieve the positive effects of PBL, it is important to respect the principles of 
good PBL when planning the course organisation. This section describes the IT service 
management undergraduate course that was offered at the University of Zagreb, Faculty of 
organization and Informatics during the 2014-2015 academic year and attended by 140 
regular students. At the end of the course, 115 students fulfilled all the course requirements 
although not all of them achieved the minimum score for a passing grade, which was 51 out 
of 100. The course description includes the main curriculum elements (Figure 1): its aims, 
intended learning outcomes, course content, learning and teaching methods and assessment, 
each of which is described in more detail in this section. 
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Assessment 

• Peer assessment 

• Self-assessment 
• Course evaluation 


3 


Course aim 

• Simulation of 
working 
environment 


Learning and 
teaching 
methods 
• PBL 


Learning 
outcomes 
• e-CF3.o* 



• ml v3** 


* European e-Competence Framework 3.0. 

** Information Technology Infrastructure Library (ITIL) framework v3 


Figure 1. IT service management curriculum elements. 

Course aim 

The course’s main aim was to teach students the management approach to IT service 
development, including not only the technical point of view but also the business perspective. 
The course was organised into 30 hours of lectures and 30 hours of laboratory exercises over 
a 17-week period. The main idea behind organising the course exercises was to put students in 
a similar environment to the one experienced in the real professional world, through the 
simulation of IT service development and communication processes with potential clients. 
The students were divided into teams of three or four members, each team representing a 
small IT company (‘virtual company’) with a task to develop an innovative IT service. In 
(Pazur Anicic and Mekovec 2014), we reported about the students’ positive attitudes towards 
this kind of course organisation, as well as their increased motivation associated with the 
work in this type of learning environment. At the end of the course students were expected to 
deliver the prototype of the mobile or web application as the final product of their work, 
together with the related business documentation. 

Course content, learning outcomes and performance plan 

The course content was organised in line with the well-known Information Technology 
Infrastructure Library (ITIL) framework based on best practices in IT service management, 
for the governance of IT and the management and control of IT sendees (Cartlidge et al., 
2011). The ITIL v3 service life cycle encompasses five stages: service strategy, service 
design, service transition, service operation and continual service improvement, which were 
used as guidelines for the development of the IT services performed by the students’ virtual 
companies. The main learning outcomes of the course exercises were to enable students to: 
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1) work in a virtual company and define/perform all necessary steps in planning the new 
IT service, 

2) prepare and participate in business meetings with potential clients and 

3) design and present a prototype of their IT service. 

Owing to the defined learning outcomes, the exercises were based on only two phases of the 
ITIL service life cycle - service strategy and service design. Each week, the course started 
with a two-hour lecture, introducing students to the theoretical knowledge related to the tasks 
and assignments to be performed during the upcoming exercises. To show the students the 
usefulness of certain skills and knowledge obtained for their potential professional profiles, 
the learning outcomes for each task within the exercises were connected to the European e- 
Competence Framework 3.0. (e-CF) (European Comitee for Standardization (CEN), 2014), as 
well as to generic competencies (Lokhoff et al., 2010). 

With only goals and guidelines in front of them, through the implementation of PBL in the IT 
service management course, the students were encouraged to develop their generic and 
transferable skills, desired to fulfil the needs of the 21 st -century job market. The teacher’s role 
in this process was to moderate and direct the students’ work to achieve the set goals. Most of 
the work was organised to be carried out during the 2-hour laboratory exercise each week 
since the opportunity to collaborate face-to-face within teams was necessary for a successful 
project output. Additionally, because ICT was perceived as useful for supporting students’ 
teamwork (Heo et al., 2010), their collaboration was supported by the learning management 
system (LMS) Moodle, which contained all the materials necessary for a successful project 
output, as well as provided a “collaboration corner” for each team. 

Teaching methods using PBL approach 

To implement PBL concepts in the course, it was necessary to consider whether the idea 
would meet the requirements of a good PBL problem. According to (Duch, 2001), a good 
PBL problem should: 

1) engage students’ interest and motivate them to understand the context of the problem, 

2) require students to make decisions/judgements based on the introduced principles or 
information, 

3) be complex enough that a solution requires the co-operation of all team members, 

4) include open-ended questions at the first stage of a problem and/or draw all team 
members into the discussion and 

5) incorporate the course’s content objectives into the problem, with a mandatory 
connection of previous knowledge to new concepts, as well as new knowledge to the 
concepts in other courses. 

Furthermore, a good PBL problem (Lohman, 2002) has the following characteristics: 
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1) The problem should be unclear, and the information needed to solve it should be 
incomplete. 

2) There should be more than one way to solve the problem. 

3) The problem should not have a single right answer. 

Another recommendation is that it should be intentionally complex, ill-structured, open-ended 
and have a holistic view of the problem or situation (Kolodner, Hmelo, & N. Hari, 1996). 
Moreover, collaboration of students or groups of students is a necessity, and the problem 
should also define the students’ roles and goals. The positive aspects of a good PBL can be 
summarised as follows: “An added benefit, if problems fit the criteria listed, is that, in 
general, they will require students to integrate knowledge from across multiple disciplines ” 
(Kolodner et al., 1996). Table A.l in Appendix 1 lists the characteristics that the relevant 
literature associates with a good PBL problem, explaining the implementation of each 
characteristic in the IT service management course. 

Assessment of students ’ work and course in general 

The students’ work was evaluated through two different types of activities during their course 
work: 1) 60 points for the individual knowledge obtained, as assessed through three tests 
during the semester, and 2) 40 points for teamwork, earned through project work. Altogether, 
five performance measurement attributes had effects on the students’ final grade, namely, 
Test 1 (Tl), Test 2 (T2), Test 3 (T3), project work - phase 1 (PI) and project work - phase 2 
(P2). The students could also obtain 20 points for self-assessment (SA) and 20 points for peer 
assessment (PA), which were excluded from the final grade and used only as an experimental 
method for using SA and PA in interactive learning environments, which students were aware 
of. At the end of the semester, the students evaluated the results of their own work (SA) in the 
second project phase (presented as an application prototype in the form of a video through 
Wiki pages in Moodle). They also evaluated the results of their peers’ work (PA), using the 
same criteria implemented in the form of rubrics in Moodle. Each student individually was 
randomly assigned two other projects to evaluate - they knew whose project they are 
evaluating, but they were also aware that the evaluated students would not know who of their 
peers have evaluated their work. The idea behind involving SA and PA was to determine how 
the scores obtained for both correlate with the students’ performance in the project work, as 
assessed by the teacher, and with their individual success in theoretical knowledge tests. The 
relevant literature shows some work on the use of SA and PA by university student 
colleagues, stressing the issues that teachers deal with when including SA and PA as part of 
formal assessment. Regarding the positive aspects, PA activities are found to stimulate 
students to initiate critical thinking and provide objective judgements about the quality of the 
work being evaluated, as well as to offer constructive comments about possible improvements 
in future work (Yu & Wu, 2011). On the other hand, PA demands considerable time and 
effort from the teacher to organise and manage the assessment process (Chen, 2010). During 
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the entire teaching process, monitoring by teachers is important to avoid particular members’ 
low level of collaboration and involvement in teamwork (Popescu, 2012). In fact, the efforts 
put into the project are not always shared equally by all team members. Other authors have 
recognised this problem in the existing literature on teamwork; consequently, they emphasise 
the need for improving the assessment methods in PBL (Fernandes, 2014). 

Aside from the assessment of students’ work, their satisfaction is an essential issue that 
should be considered when the perception of course quality or course content, as well as 
course effectiveness, is evaluated. The results presented by (Liaw & Huang, 2013) indicate 
that the learning activity could be affected by environmental characteristics, environmental 
satisfaction and learner characteristics. Therefore, we have also incorporated individual 
students’ evaluation of different course aspects at the end of the semester, using the self- 
created semantic differential scale. 

The methods and results of student performance and course satisfaction, both related to the 
assessment phase in the curriculum design, are described in the next section. 


METHOD 


Student satisfaction 

At the end of the course, the students were given the opportunity to anonymously express 
their satisfaction with the laboratory exercises organized according to the PBL principles. A 
total of 123 students, of which 107 (87%) were male and 16 (13%) were female, filled the 
questionnaire about their satisfaction with certain course aspects. Most of the students (66%) 
stated that their initial interest in this course was medium, 21% assessed their interest as high, 
and 13% considered it low. The students were also asked to estimate the effort they put into 
the course completion on a five-point Likert-type scale (from l=minimum effort to 
5=maximum effort). Only two students (1.6%) assessed their effort as minimal, and 10.6% 
considered it very low. Most of the students (44%) stated their effort as medium, and another 
44% reported it as high (32.5%) or very high (11.3%). 

For the assessment of the students’ general satisfaction with some aspects of interactive 
learning environment implemented within the laboratory exercises of the IT service 
management course according to the PBL principles, we used a self-created scale similar to 
the one proposed by (Lin, 2008), consisting of (1) several general questions showing the 
students’ attitude towards the course and the work performed and (2) the semantic differential 
scale with a five-point Likert-type scale for the assessment of six aspects of the learning 
environment. The semantic differential scale, a technique for evaluating people on their 
responses to pairs of bipolar adjectives in relation to concepts, has been found to be 
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appropriate for the evaluation of experiential teaching methods (Whitney & Soukup, 1988). 
Different aspects of the learning environment were assessed, using the same semantic 
differential scale with the following eight pairs of attributes: usual-innovative, boring- 
interesting, motivating-demotivating, challenging-unchallenging, low quality-high quality, 
appropriate-inappropriate, unnecessary-necessary and useless-useful. The questionnaire was 
first piloted in the same course during the 2013-2014 academic year (Pazur Anicic and 
Mekovec, 2014), when many PBL elements were already implemented. 

Table A.l. brings the connection between characteristics of a good PBL problem and their 
implementation in IT service management course. With the analysis of student satisfaction we 
wanted to see how satisfied are students with the course implemented according to PBL 
principles in the following aspects of learning environment: 

1) content of laboratory exercises (related to the ITIL v3 concept), 

2) teaching methods (teamwork - each team representing a virtual IT company, students 
given basic instructions only for task completion, the teacher with the role of 
instructor and only providing guidance to students on their tasks), 

3) simulation of work environment (business meeting, contracting and maintenance, 
assigning roles in the company, determining job vacancies within the company), 

4) collaboration with employers (visiting lecturers from several IT companies, practical 
examples of potential project implementation, presentation of students’ final projects 
in front of company representatives), 

5) future career development (insight into the labour market, meeting the standards of 
the profession - e-CF 3.0., preparation of motivation letter) and 

6) Moodle (usage of this LMS to support all course activities). 

Students were not expected to evaluate the implementation of certain PBL principle in the 
course, as it is something they are not familiar with. But they indirectly evaluated it through 
expressing their satisfaction with six aspects of learning environment which are directly 
affected with PBL principles. Those aspects refer directly to the implementation of PBL 
principles, but also stimulate students to think about the connection of this type of learning 
activities with the world of work and the impact to their future career development. 

Table C.l in Appendix C shows the average values, standard deviation and Cronbach’s alpha 
coefficient for each of the six evaluated aspects of the learning environment. The Cronbach’s 
alpha coefficient for the internal consistency of the scale for each aspect, assessed by eight 
pairs of attributes, is quite acceptable, ranging from 0.868 to 0.926. Figure 2 shows that the 
students’ assessments of all aspects of the learning environment are on the right, positive side 
of the semantic differential scale. The students gave the highest ratings to collaboration with 
employers (M=4.438, sd=0.603) and simulation of the work environment (M=4.137, 
sd=0.722) in the regular exercises. All the other aspects were rated above average, showing 
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the students’ positive attitudes towards the content of the exercises that were structured in line 
with the ITIL best practices (M=3.836, sd=0.67), methods used in the teaching process in line 
with PBL (M=3.776, sd=0.747), activities connected with their future career development and 
labour market needs according to e-CF 3.0 (M=3.830, sd=0.745), and usage of Moodle to 
support teaching activities and the students’ group work (M=3.640, sd=0.779). 


Usual 
Boring 
Demotivating 
Unchallenging 
Low quality 
Inappropriate 
Unnecessary 
Useless 



1 2 3 4 5 

— Content 
-II-Methods 

— -A-Simulation 

— •• — Employee collaboration 

. + .Careers 


Innovative 

Interesting 

Motivating 

Challenging 

High quality 

Appropriate 

Necessary 

Useful 


Figure 2. Students’ assessment of different course aspects. 

To gain better insights into the results, we analysed the effects of the students’ initial interest 
in the course and their efforts in the course completion on the six aspects of the learning 
environment that were of interest. Table C.2 in Appendix C presents the descriptive statistics 
for the dependent variables (aspects of the learning environment). Generally, it could be 
observed that the students with lower initial interest in the course assessed all the course 
aspects with slightly lower values. The situation was similar with the students who put less 
effort into performing the course tasks although it was not always the case for the students 
who rated their efforts 3 points or higher on the five-point Likert-type scale (from 
l=minimum effort to 5=maximum effort). To determine if there were statistically significant 
differences in the effects of the initial course interest and work effort among different groups, 
we performed the analysis of variance (ANOVA). The test scores indicated no statistically 
significant differences among the groups regarding the effects of different levels of initial 
interest and work effort, except in a few cases. There was a significant effect of work effort on 
the students’ assessment of the teaching methods used in the course (df=4, F=3.121, 
p=0.0179), showing that those who put less effort into the course assignment completion 
reported lower satisfaction with the teaching methods. Initial interest (df=2, F=15.336, 
p=1.33e-06) and work effort (df=4, F=3.373, p=0.0121) both showed a significant difference 
in the students’ attitudes towards the aspect of their future career development, indicating that 
those with lower interest and work effort did not perceive the importance of their future career 
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at this stage of their study. The students’ interest in the course also had a significant effect on 
their attitudes towards the use of Moodle in the teaching process (df=2, F=5.675, p=0.00451). 

Work performance patterns 

Besides students’ satisfaction with the particular aspects of learning environment set 
according to PBL principles, another research question is related to the identification of 
students’ work performance patterns in the introduced PBL environment. To identify different 
groups of students in the set learning environment (in terms of their work performance), the 
hierarchical clustering algorithm was applied in the statistical package R. Five performance 
measurement attributes were used (Tl, T2, T3, PI and P2) for clustering, amended with the 
SA and PA results. The clustering algorithm resulted in seven clusters (C1-C7), as shown in 
Figure 3, with the descriptive data (mean and standard deviation) in Table B.l in Appendix B. 


o> 

<D 


I 



Figure 3. Clusters based on students’work performance. 

Cluster 1 

Cluster 1 consisted of 18 students with an overall score (M=55.78, sd=5.32) lower than 
average (M=65.10, sd=9.18). The students included in this cluster had lower performance in 
individual tests (Tl, T2 and T3), as well as project work (PI and P2) results that were slightly 
lower than average. Their peers assessed (PA) their project work as lower than average 
(M=8.67, sd=3.09), but they evaluated their own work (SA) as much higher (M=13.67, 
sd=2.03) than their peers and teachers did (M=12.06, sd=2.51). Better scores in T3 indicated 
the students’ intention to obtain enough points for a passing (or higher) grade at the end of the 
semester. 

Cluster 2 

Cluster 2 comprised 21 students with a lower than average final score (M=60.51, sd=5.75) but 
higher than that in Cluster 1 (M=55.78, sd=5.32). The students in this cluster obtained lower 
than average test results in all three tests (Tl, T2 and T3), near average project results for the 
first phase (PI) and above average scores for the final project (P2) phase. These outcomes 
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indicated that the students in this group were more practically than theoretically oriented, but 
their general performance was quite low. They seemed aware of the low efforts they put into 
the course tasks because their SA results (M=13.57, sd=2.50) were lower than those of the PA 
(M=14.31, sd=1.61) and their P2 grades received from the teachers (M=14.43, sd=3.84). 

Cluster 3 

In Cluster 3, 12 students obtained a higher than average overall score (M=73.09, sd=6.36). 
This group’s main characteristic was that the students assessed their final project work (SA) 
with much higher grades (M=18.08, sd=2.53) than those given by their peers (PA) (M=14.88, 
sd=2.55) and teachers (P2) (M=14.50, sd=2.96). For the members of this group, the negative 
correlation between the first two tests (T1 and T2) and the project work results (PI and P2) 
showed that the students were either practically oriented and thus put more effort into the 
project work than in the theoretical tests (which would explain their high SA scores) or were 
assigned to teams in which some members worked more on the project results. Their above 
average results (M=16.09, sd=1.54) in the last test (T3) indicated their efforts to obtain a 
better final grade towards the end of the course. 

Cluster 4 

For the 17 students included in Cluster 4, their overall course score was above average 
(M=60.57, sd=6.98), and their project work was much better assessed in the second phase 
(P2). On the other hand, their test results were below average for all three tests (Tl, T2 and 
T3), with very low results in the second test (T2) (M=9.41, sd=2.46). The higher score 
obtained in the final project phase clearly showed that the students had been working harder 
on their project in the second phase. However, they seemed unrealistic in the SA of their own 
work (M=17.94, sd=l.89), which was higher than those obtained from their teachers (P2) 
(M=15.18, sd=2.46) and peers (PA) (M=16.82, sd=2.51). 

Cluster 5 

The main characteristic of the 22 students in Cluster 5 was their achievement of higher scores 
in individual knowledge tests (except T2) than in project teamwork (PI and P2), which 
differentiated them from the students in the other six clusters. The SA (M=13.05, sd=1.17) 
and PA (M=14.09, sd=2.66) of their final project work were similar to their teachers’ 
assessment (P2) (M=13.96, sd=2.17). Of all the clusters, this group’s overall course success 
(M=64.82, sd=5.91) was closest to the average course success for all clusters (M=65.10, 
sd=9.18). 


Cluster 6 

Cluster 6 consisted of 13 students with an above average overall score (M=71.58, sd=3.98) 
and higher scores in project work (PI and P2) than in individual theoretical knowledge (Tl, 
T2 and T3). The fact that the students evaluated their work (SA) lower than their teachers did 


26 



K. P. Anicic, R. Mekovec 


JPBLHE: Early view, 2016 


implied that either these students put less effort into teamwork than the other team members 
did and were aware of it, or they put more effort into project work than the tests and were 
very self-critical. Both their PA (M=13.50, sd=1.94) and SA (M=15.39, sd=2.76) evaluations 
were lower than their teachers’ (P2) (M=18.23, sd=1.42). It is important to note the really low 
scores obtained from their peers, which is hard to interpret. 

Cluster 7 

Cluster 7 included 12 students showing the highest course scores (M=79.10, sd=4.44), with 
higher than average results for four of the five assessment aspects, except T2, which was 
slightly below average. From the much lower SA results (M=13.00, sd=3.49), which differed 
significantly from the PA (M=16.54, sd=2.37) and the teachers’ assessment (P2) (M=17.25, 
sd=2.09), it could be concluded that the students in this group were very self-critical. The 
possible explanation for this case was that they put more effort into theoretical foundations, 
not only into project work, giving them a broader theoretical background about the problem 
they were solving and making them more conscious about possible improvements in their 
own work. 


DISCUSSION AND CONCLUSION 

In this paper, we presented a framework for introducing PBL principles in the IT service 
management course. One of the major changes introduced to the PBL elements in the course 
is connected with teachers’ and students’ roles (Roberto & Ribeiro, 2008) in this new form of 
an interactive learning environment. Teachers are responsible for guiding students in their 
process of problem solving and therefore assume the role of consultants, as well as facilitators 
of students’ learning. For their part, students are responsible for learning and proposing 
solutions to the introduced problem. This discussion is organised around several main 
findings referring to the research questions related to different patterns of the students’ work 
performance and their satisfaction with the introduced learning environment, taking into 
account this study’s limitations. 

First, let us discuss the answer to the first research question: How do students assess 
particular aspects of this set learning environment, organised according to problem-based 
learning principles? The answer to this research question was obtained by the use of semantic 
differential scale for the assessment of student satisfaction with six course aspects: 1) content 
of laboratory exercises, 2) teaching methods, 3) simulation of work environment, 4) 
collaboration with employers, 5) future career development and 6) usage of this LMS Moodle 
to support all course activities. The general conclusion about the students’ satisfaction with 
the six learning environment aspects, organised according to PBL principles, was found to be 
very positive. It is interesting to see that students were most satisfied with the collaboration 
with employers (M=4.438, sd=0.603) and simulation of the work environment (M=4.137, 
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sd=0.722) within the course, which indicates their positive attitude to the connection with the 
world of work within regular courses. We can connect our research results of students’ 
satisfaction with certain aspects of learning environment organized according to PBL 
principles with the positive results of some previous researches on PBL implementation. For 
example, (Popescu, 2012) showed that, from the students’ point of view, PBL contributes to 
improvements in terms of learning, motivation, enjoyment, involvement, teamwork quality 
and overall satisfaction but increased their study time and effort compared to traditional 
learning. This can be related to the significant effect of work effort on the students’ 
assessment of the teaching methods used in our course (df=4, F=3.121, p=0.0179), showing 
that the students who put less effort into the course assignment completion expressed a lower 
level of satisfaction with the teaching methods. 

Second, within the implementation of the PBL environment, we wanted to research the 
different patterns in the students’ work performance and the SA and PA characteristics 
through the rubrics in the Moodle workshop. That was researched by the application of cluster 
analysis and provides an answer to the research question: Is it possible to identify specific 
patterns in students ’ work performance in the established problem-based learning 
environment? Cluster analysis resulted in seven different groups of students according to their 
work performance. Although both teachers and students were introduced to increased 
time/workload needed for the adaptation to the new learning environment within laboratory 
exercises organized according to PBL principles, it is noteworthy that the average scores for 
all three tests (Tl=12.56, T2=10.61 and T3=13.53) were lower than those for the project work 
results (Pl=14.72 and P2=15.09). These results indicated the students’ positive attitude 
towards project-based teamwork in solving real-world problems compared to individual 
theoretical knowledge. As the project-based learning organized according to PBL principles is 
also based on some theoretical foundations (in this concrete case it is the ITIL v3), it would be 
interesting to further research if students also better learn and gain theoretical knowledge from 
this type of activity, comparing to the classical ex-cathedra lectures. If further research shown 
that this kind of problem-solving teamwork contributes to the development of the necessary 
skills for the students’ future workplaces, both theoretical knowledge and practical skills, it 
should be considered that the practical part of the course work is weighted with a stronger 
factor in the course’s overall point structure than the strictly theoretical knowledge tests. 

Besides providing an answer to the second research question, cluster analysis also provided 
some insight into the application of student self-assessment and peer-assessment in the 
courses organized according to PBL principles. The results on the use of SA and PA also 
indicated different patterns in student behaviour. Although not directly addressed in the 
research questions of this paper, we will refer shortly to these results as well, since they 
indicate a potentially interesting area for further research on the assessment within PBL as 
one of the main curriculum elements described in the theoretical part. In other research 
studies, using rubrics in the portfolio assessment showed a significant difference between SA 
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and teacher assessment but no significant difference between SA and PA (Chang, Tseng, & 
Lou, 2012). Generally, our results found similar values for P2 (M=12.35, sd=3.23) and SA 
(13.37, sd=2.48), while PA was much lower (M=10.29, sd=2.92) but with differences from 
cluster to cluster. This confirmed previous findings that PA failed to demonstrate acceptable 
validity if the teacher assessment was used as a valid exterior criterion (Chang et al., 2012), 
and it could not be used as a reliable assessment method (Chang, Tseng, Chou, & Chen, 
2011). For example, for Clusters 1 and 3, PA was much lower than P2. It can be concluded 
that PA and SA are recognised as positive and central techniques to develop critical aptitudes 
in students (Wright, 2011), but they require control from the tutor (Martinez-Gonzalez & 
Duffing, 2007). On the other hand, self-evaluation points out two main patterns: 1) students 
who assess their own work higher than its realistic score, such as in Clusters 1, 3 and 4, and 
quite the opposite, 2) students who are really self-critical and rate their work much lower in 
comparison with their comprehensive work performance, similar to those in Cluster 7. 

The main limitation in our approach was the anonymous questionnaire with the semantic 
differential scale, so we were unable to observe the relationships between the students’ 
satisfaction with the set learning environment in line with PBL principles and their work 
performance. In future research, it would be interesting to find out how students’ initial 
interest and work effort correlate with their work performance. We used the anonymous 
questionnaire mainly because we wanted to obtain honest responses from the students, which 
would have been a problematic issue if we had asked them to write their names in the 
completed questionnaires before they obtained their final course grades. 

As for the conclusion, we would like to refer to the introduction, which indicated the 
importance of the implementation of new teaching methods in courses, in order to better 
prepare students for the world of work. One of the main roles of higher educational 
institutions is to educate future professionals in a certain field and to provide them with the 
knowledge and skills desired by their future employers. The existing literature has shown that 
the interactive PBL environment offers the possibility to motivate and promote students’ 
awareness and responsibility for their learning, with respect to the development of their 
professional skills and attitudes. Our work represents a framework for implementing PBL 
principles into IT service management course that could serve as guidelines for other 
practitioners willing to implement PBL principles in their courses. The interactive PBL 
environment positively influences students’ self-directed learning and could therefore be used 
to narrow the gap between the student profiles desired by business organisations and the 
profiles currently offered by higher educational institutions. Except describing in detail 
teaching methods organized according to the PBL principles, this study’s results contribute to 
the last phase of the curriculum cycle, the assessment phase, by indicating the potential for the 
use of self-assessment and peer-assessment methods. 

The presented research contributes to several areas of improvement in teaching practices. 
First, gaining insights into student satisfaction with particular course elements can be helpful 
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in the further innovation of a particular course or study programme. In this way, students can 
benefit from modern and creative courses that simulate the actual work environment to solve 
real-world problems. They can also leam about some techniques to be used in their future 
professional work. Furthermore, the paper contributes to the individualised approach in 
course development and delivery, based on the recognition of different students’ work 
performance patterns. When teachers possess information about the patterns in students’ work 
performance in the set learning environment, they can adapt certain course elements to the 
needs of specific groups of students. This strategy enhances creativity, critical thinking and 
the development of other transferable and employability skills. Consequently, the results of 
this research offer valuable inputs in the necessary process of narrowing the gap between 
current students’ skills (developed during education) and the skills desired by employers. 
Therefore, this study’s findings could be used by both researchers and practitioners in their 
efforts to create or further research a stimulating, interactive learning environment based on 
PBL that improves the preparation of students for their future workplaces. 


Appendix A. Description of PBL elements 

Table A.l. Characteristics of a good PBL problem and their implementation in IT service management 
course. 


Characteristic of PBL 
problem introduced in IT 
management course 

How was it implemented? 

The problem engages students’ 
interest and motivates them to 
understand the context of the 
problem (Duch, 2001; 

Kolodner et al., 1996) 

Organising the course exercises aimed to present students with real- 
world situations in which they would be active participants. 
Students attending our faculty and this course would be employed 
in the information and communication a low grade in technology 
(ICT) domain. Therefore, new knowledge about the organisation 
and management of IT companies would be welcomed upon 
completion of their education process and during their job search. 
Moreover, students were encouraged to choose a problem according 
to the teams’ previous knowledge and interests to make them more 
motivated to work on its solution. 

The problem requires students 
to make decisions/judgements 
based on introduced principles 
or information (Duch, 2001). 

In the beginning of every exercise, students were introduced to the 
set goals. Furthermore, possible ways of achieving defined goals 
were presented, and students were requested to consult particular 
information sources to learn more about how to conduct a particular 
analysis or compilation, for example. 

Additionally, during the organised lectures by invited speakers from 
different IT companies, students were introduced to real problems 
occurring in such companies on a daily basis or at a particular phase 
of service development. This was an opportunity to familiarise 
themselves with the development of services based on presented 
practice. Consequently, they were able to apply the obtained 
information in their own virtual companies. 

The problem is complex 
enough that a solution requires 
the co-operation of all 

Owing to the time limitation, the tasks connected to the particular 
exercises, as well as the overall course goals, were too complex to 
be solved by one student during a given time period (each exercise 
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members of the students’ teams 
(Duch, 2001; Kolodner et al., 
1996; Weiss, 2003) 

lasted for 90 minutes, performed 14 times during the semester). To 
complete all necessary activities, students were required to 
communicate within the group and to work as a team. 

The problem includes open- 
ended questions (Duch, 2001); 
(Kolodner et al., 1996). 

During the first week of the semester, the student teams should have 
agreed on the type of company they would operate and the new IT 
service they would introduce to the marketplace. For a successful 
decision, this process required the consensus of all team members. 
All team members were also expected to be familiar with all the 
defined processes in their virtual company. 

Dealing with open-ended questions was most pronounced during 
the simulation of business meetings with the potential clients, which 
were organised two times during the semester. The purpose of these 
meetings was to introduce students to common business 
communication, as well as to prepare them to different aspects of 
service presentation. During the meeting, the students played their 
respective roles in their company and were required to negotiate 
with a new client about questions from their own domain, without 
having previously prepared the list of questions to be discussed with 
a potential client. 

The problem is unclear 
(Lohman, 2002); in some 
studies, this type of problem is 
defined yet ill-structured 
(Baturay & Bay, 2010; 
Kolodner et al., 1996; Weiss, 
2003) 

As already mentioned, students were encouraged to choose a 
problem on their own, and the teachers’ role was to judge if the 
problem would be adequate to satisfy all the learning outcomes of 
the course. Thus, the problem was not based only on a particular 
discipline and previously defined. In the beginning of each exercise, 
students were provided with information regarding the goals of 
particular exercises, the connection with previous work and the 
connection with the overall goals of the course. Students were 
invited to apply their current knowledge, such as implementing a 
SWOT analysis or defining the graphical standard of a mobile 
application. They were also introduced to tools, methods and 
techniques that could be used for performing the exercise tasks and 
attaining the goals. They were not given a definition of a specific 
output since there was no ideal solution for each project. 

There should be more than one 
way to solve the problem 
(Lohman, 2002). The problem 
should have several 
interpretation/solutions 
(Baturay & Bay, 2010; 

Kolodner et al., 1996) 

During each exercise, students were introduced to a new element of 
service development. Various methods, techniques and tools that 
could be used to efficiently complete defined tasks were presented. 
The student teams could choose how they would act and which 
methods and tools they would use; therefore, various solutions to 
the same task were possible. 

The information provided is 
incomplete (Lohman, 2002). 

During the first business meeting, students presented their ideas on 
their new IT service to sell it to the potential clients (their teachers). 
In this negotiating process, the sellers (students) attempted to 
capture all the client requirements. The clients were demanding 
various modifications of the presented product/service. Before the 
meeting, students were informed about its puipose only, but they 
were unfamiliar with their potential clients’ identities and 
requirements. Moreover, during the meeting, students were required 
to be very careful about how they would react to client demands 
because their task in the second cycle of the exercises was to 
implement and deliver their final service prototype according to 
what was negotiated. 

The connection of previous 
knowledge to new concepts, as 
well as new knowledge to the 

To solve a particular task, students needed to use their previous 
knowledge. New concepts, which were expected to be learned 
through a particular exercise, were referenced on students’ previous 
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concepts in other courses, is 
mandatory (Duch, 2001; Weiss, 
2003) 

knowledge, as well as on possible future usage during their 
graduate studies. 

For example, students were introduced to the scope of the SWOT 
analysis during their previous work in other courses, and in this 
course, they were required to apply the SWOT analysis to their 
virtual company that was placed in a concrete business 
environment. On the other hand, the final prototype produced 
during this course could be used as input for further work in other 
courses (e.g., mobile application development). 

Teachers are only facilitators 
and consultants; the students’ 
role is to work on the problem 
(Baturay & Bay, 2010). 

Teachers were not using the usual techniques, such as presentations 
or the conference method, during the exercises, but students were 
provided with theoretical knowledge during the lectures. Instead, 
teachers were only giving instructions to students on how a 
particular task could be solved, and their primary role was to 
discuss with students about their problems in the company and 
future steps in their service development. In this manner, teachers 
assumed the consultant’s role for all virtual companies and helped 
students examine the problem with a more realistic and business 
view. 

Students are motivated towards 
lifelong and self-directed 

learning (Weiss, 2003); they 
evaluate their own work 
(Baturay & Bay, 2010). 

One of the activities carried out during the semester was the 
analysis of current job offers on the real ICT market. Students had 
the task of comparing their current knowledge and skills with those 
recurring in job profiles. In this way, students were motivated to 
make plans regarding their future professional career. They were 
also encouraged to assess their own work and compare it to those of 
their peers through self-assessment and peer-assessment activities. 
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APPENDIX B: Results of clustering 


Table B.l. Descriptive statistics for all clusters 



Overall 

Cl (18) 

C2 (21) 


C3 (12) 

C4 (17) 

C5 (22) 

C6 (13) 

C7 (12) 

Element 

Mean 

Std 

Mean 

Std 

Mean 

Std 

Mean 

Std 

Mean 

Std 

Mean 

Std 

Mean 

Std 

Mean 

Std 

T1 

12.56 

3.49 

10.19 

2.20 

9.74 

2.60 

14.88 

3.57 

12.21 

2.28 

14.36 

2.34 

11.27 

1.90 

15.29 

2.24 

T2 

10.61 

3.04 

7.47 

2.67 

9.45 

1.75 

10.71 

3.17 

9.41 

2.46 

10.21 

1.42 

13.39 

1.34 

13.63 

2.60 

T3 

13.53 

3.62 

13.73 

2.58 

12.23 

2.02 

16.09 

1.54 

10.84 

1.91 

14.11 

1.42 

12.85 

1.65 

14.85 

2.57 

PI 

14.72 

3.21 

12.33 

3.50 

14.67 

2.10 

16.92 

2.63 

13.00 

3.84 

12.18 

3.54 

15.85 

2.91 

18.08 

1.56 

P2 

15.09 

3.23 

12.06 

2.51 

14.43 

3.84 

14.50 

2.96 

15.18 

2.46 

13.96 

2.17 

18.23 

1.42 

17.25 

2.09 

PA 

14.12 

2.92 

8.67 

3.09 

14.31 

1.61 

14.88 

2.55 

16.82 

2.51 

14.09 

2.66 

13.50 

1.94 

16.54 

2.37 

SA 

14.96 

2.48 

13.67 

2.03 

13.57 

2.50 

18.08 

2.53 

17.94 

1.89 

13.05 

1.17 

15.39 

2.76 

13.00 

3.49 

Overall 

course 

65.10 

9.18 

55.78 

5.32 

60.51 

5.75 

73.09 

6.36 

60.57 

6.98 

64.82 

5.91 

71.58 

3.98 

79.10 

4.44 


score 


APPENDIX C: Evaluation of different course aspects 


Table C.l. Descriptive statistics of students' satisfaction with different course aspects 


Content Method Simulation Employee Career Moodle 



Mean 

SD 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Usual - Innovative 

3.992 

0.835 

3.959 

0.936 

4.285 

0.835 

4.642 

0.642 

4.041 

0.872 

3.870 

0.941 

Boring - Interesting 

3.683 

0.961 

3.829 

0.894 

4.146 

0.929 

4.512 

0.761 

4.065 

0.894 

4.098 

0.900 

Demotivating - Motivating 

4.146 

0.856 

4.171 

0.786 

4.285 

0.741 

4.528 

0.631 

4.114 

0.812 

3.992 

0.864 

Unchallenging - Challenging 

4.057 

0.771 

4.016 

0.839 

4.081 

0.836 

4.390 

0.785 

3.886 

0.851 

3.829 

0.989 

Low quality - High quality 

3.764 

1.064 

3.520 

1.089 

4.098 

0.962 

4.154 

0.984 

3.577 

0.950 

3.407 

0.931 

Inappropriate - Appropriate 

3.488 

1.011 

3.569 

0.933 

4.106 

0.847 

4.545 

0.668 

3.724 

1.011 

3.366 

1.010 

Unnecessary - Necessary 

3.642 

1.017 

3.553 

1.026 

4.098 

0.962 

4.480 

0.813 

3.650 

1.008 

3.301 

1.032 

Useless - Useful 

3.919 

0.845 

3.593 

1.055 

4.000 

0.950 

4.252 

0.874 

3.585 

0.958 

3.260 

1.055 

Cronbach's Alpha 

0.868 

0.910 

0.926 

0.903 

0.922 

0.921 
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Table C.2 Means and SDs of dependent variables 




Interest 


Work 



Mean 

St.Dev 


Mean 

St.Dev 

Content 

1 

3.555 

0.428 

1 

3.063 

0.442 


2 

3.849 

0.658 

2 

3.596 

0.636 


3 

3.971 

0.790 

3 

3.734 

0.667 





4 

4.009 

0.613 





5 

4.071 

0.737 

Method 

1 

3.492 

0.543 

1 

2.625 

0.000 


2 

3.792 

0.751 

2 

3.490 

0.658 


3 

3.904 

0.820 

3 

3.757 

0.708 





4 

3.997 

0.701 





5 

3.652 

0.903 

Simulation 

1 

3.844 

0.691 

1 

3.438 

0.088 


2 

4.117 

0.705 

2 

4.019 

0.775 


3 

4.380 

0.739 

3 

4.007 

0.747 





4 

4.341 

0.630 





5 

4.268 

0.747 

Employee 

1 

4.375 

0.631 

1 

3.563 

0.265 


2 

4.471 

0.521 

2 

4.625 

0.445 


3 

4.375 

0.811 

3 

4.317 

0.691 





4 

4.541 

0.528 





5 

4.563 

0.424 

Career 

1 

3.219 

0.693 

1 

3.375 

0.530 


2 

3.790 

0.675 

2 

3.663 

0.035 


3 

4.332 

0.675 

3 

3.727 

0.709 





4 

4.088 

0.629 





5 

3.714 

0.797 

Moodle 

1 

3.172 

0.929 

1 

3.188 

0.619 


2 

3.628 

0.711 

2 

3.288 

0.979 


3 

3.966 

0.759 

3 

3.567 

0.681 





4 

3.825 

0.748 





5 

3.786 

0.953 


Legend: For the variable Interest, l=small, 2=medium, 3=high 
For the variable Work, 1= minimum, 2—very small, 3= medium, 4= high, 5—maximum 
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